Fix keyvault extension authentication to always use a user managed identity by tamirkamara · Pull Request #3492 · microsoft/AzureTRE

tamirkamara · 2023-05-09T08:28:15Z

What is being addressed

The keyvault VM extension sometimes fails as it didn't have a config to always use the assigned user managed identity.

How is this addressed

Add config to set this.

github-actions · 2023-05-09T08:29:05Z

Unit Test Results

0 tests 0 ✔️ 0s ⏱️
0 suites 0 💤
0 files 0 ❌

Results for commit ea93b35.

♻️ This comment has been updated with latest results.

tamirkamara · 2023-05-10T16:03:35Z

/test-shared-services

tamirkamara · 2023-05-10T16:05:09Z

/help

tamirkamara · 2023-05-11T11:14:08Z

/test-shared-services

github-actions · 2023-05-11T11:14:22Z

🤖 pr-bot 🤖

🏃 Running shared service tests: https://github.com/microsoft/AzureTRE/actions/runs/4947240687 (with refid b4903e3e)

(in response to this comment from @tamirkamara)

tamirkamara · 2023-05-11T14:24:33Z

/test-shared-services

github-actions · 2023-05-11T14:24:53Z

🤖 pr-bot 🤖

🏃 Running shared service tests: https://github.com/microsoft/AzureTRE/actions/runs/4948983697 (with refid b4903e3e)

(in response to this comment from @tamirkamara)

github-actions · 2023-05-16T03:21:12Z

🤖 pr-bot 🤖

🏃 Running shared service tests: https://github.com/microsoft/AzureTRE/actions/runs/4948983697 (with refid b4903e3e)

(in response to this comment from @tamirkamara)

github-actions · 2023-05-16T06:07:57Z

🤖 pr-bot 🤖

🏃 Running shared service tests: https://github.com/microsoft/AzureTRE/actions/runs/4948983697 (with refid b4903e3e)

(in response to this comment from @tamirkamara)

marrobi · 2023-05-16T20:19:06Z

Ok, so tested this, bundle 2.5.1.

Settings - off the VM itself:

{"runtimeSettings": 
[{"handlerSettings": {"publicSettings": {"secretsManagementSettings": 
{"authenticationSettings": {"msiClientId": "/subscriptions/1e836626-b8c0-49a5-b4aa-9e9f93e31abf/resourceGroups/rg-mrtredemo27/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-nexus-mrtredemo27", 
"msiEndpoint": "http://169.254.169.254/metadata/identity"}, 
"observedCertificates": ["https://kv-mrtredemo27.vault.azure.net/secrets/nexusvmcert"], "pollingIntervalInS": "3600",
 "requireInitialSync": true}}, "protectedSettings": null, 
"protectedSettingsCertThumbprint": null}}]}

Extension logs:

2023-05-16 19:49:22: <debug> [Global]logFolder: /var/log/azure/Microsoft.Azure.KeyVault.KeyVaultForLinux
2023-05-16 19:49:22: <debug> [Global]Starting akvvm_service
2023-05-16 19:49:22: <info> [VMExtension]Starting extension
2023-05-16 19:49:22: <debug> [CertificateManagementConfiguration]Found v1.x configuration
2023-05-16 19:49:22: <info> [CertificateManagementConfiguration]Defaulting to MSI authentication.
2023-05-16 19:49:22: <info> [KVUnixService]Checking Linux distribution and version
2023-05-16 19:49:22: <info> [KVUnixService]OS info: ubuntu 18.04
2023-05-16 19:49:22: <info> [KVUnixService]Service Running...
2023-05-16 19:49:22: <info> [CertificateManager]Entering worker loop..
2023-05-16 19:49:22: <debug> [CertificateManager]MIN_POLLING_INTERVAL_SEC: 1, RandomMS: -1000
2023-05-16 19:49:22: <info> [CertificateManager]Starting refreshing observed certificates...
2023-05-16 19:49:22: <info> [CertificateManager]Beginning refresh for: https://kv-mrtredemo27.vault.azure.net/secrets/nexusvmcert
2023-05-16 19:49:22: <info> [KeyVaultClient]Getting new auth challenge
2023-05-16 19:49:22: <debug> [UnixKeyVaultHttpClient]Using CAfile for TLS: /etc/ssl/certs/ca-certificates.crt
2023-05-16 19:49:22: <debug> [UnixKeyVaultHttpClient]Using CApath for TLS: /etc/ssl/certs
2023-05-16 19:49:22: <debug> [AuthClient]AcquireTokenCallback invoked
2023-05-16 19:49:22: <debug> [AuthClient]acquiring token
2023-05-16 19:49:22: <debug> [MSIAuthClient]acquiring token via MSI
2023-05-16 19:49:22: <debug> [MSIHttpClient]MSI URL: http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&authority=https://l
ogin.microsoftonline.com/62d053bf-6309-4727-a3bf-bf604826c289&resource=https://vault.azure.net
2023-05-16 19:49:22: <debug> [UnixKeyVaultHttpClient]Using CAfile for TLS: /etc/ssl/certs/ca-certificates.crt
2023-05-16 19:49:22: <debug> [UnixKeyVaultHttpClient]Using CApath for TLS: /etc/ssl/certs
2023-05-16 19:49:22: <error> [CertificateManager]Refreshing 'https://kv-mrtredemo27.vault.azure.net/secrets/nexusvmcert' failed with RequestExcept
ion: 403; desc: {"error":{"code":"Forbidden","message":"The user, group or application 'appid=d1e82d40-40e3-4b22-a88b-b3d4ccfc0f48;oid=a6ed1cfb-b5b4-4d63
-940d-c414424b990e;iss=https://sts.windows.net/62d053bf-6309-4727-a3bf-bf604826c289/' does not have secrets get permission on key vault 'kv-mrtredemo27;l
ocation=westeurope'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287","innererror":{"code":"AccessDenied"}}}
2023-05-16 19:49:22: <error> [CertificateManager]Failed to download one or more certificates.
2023-05-16 19:49:22: <info> [UnixCertificateManager]Checking state of termination event with a timeout of 3599000

The id a6ed1cfb-b5b4-4d63-940d-c414424b990e is the VMs system assigned identity.

The cloud-init logs show:

Setting up Nexus SSL...
Checking for nexus-data/keystores directory...
Waiting for cert to be downloaded from KV...
ERROR - Timeout while waiting!

Not sure why the VM would have system assigned identity enabled?

tamirkamara · 2023-05-18T06:07:25Z

/test

github-actions · 2023-05-18T06:07:40Z

🤖 pr-bot 🤖

🏃 Running tests: https://github.com/microsoft/AzureTRE/actions/runs/5011072398 (with refid b4903e3e)

(in response to this comment from @tamirkamara)

tamirkamara · 2023-05-18T13:48:54Z

@marrobi something is off here. I don't have any system managed identity on my nexus VM, nor is there one in the environment associated with this PR.
Are you sure you have the right bundle? I suggest maybe deleting the repo from the ACR and republishing (with a new version) to verify.

marrobi · 2023-05-19T08:59:12Z

@tamirkamara if I deploy a VM through the portal, nothing set, it gets a system ID, if I look at the ARM JSON used to deploy there isn't one set. So wonder if its a default somewhere?

marrobi · 2023-05-19T09:08:46Z

@tamirkamara I think the JSON in terraform isn't quite right:

https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/key-vault-linux#extension-schema

authenticationSettings isn't under secretManagementSettings, just under settings.

Might try give it a go.

marrobi · 2023-05-19T10:56:12Z

Ok, with that change I get a different error:

2023-05-19 10:41:21: <debug> [MSIHttpClient]MSI URL: http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&authority=https://l
ogin.microsoftonline.com/62d053bf-6309-4727-a3bf-bf604826c289&resource=https://vault.azure.net&client_id=/subscriptions/1e836626-b8c0-49a5-b4aa-9e9f93e31
abf/resourceGroups/rg-mrtredemo27/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-nexus-mrtredemo27
2023-05-19 10:41:21: <error> [MSIAuthClient]failed to retrieve MSI token from response: {"error":"invalid_request","error_description":"Identity not 
found"}
2023-05-19 10:41:21: <error> [CertificateManager]Refreshing 'https://kv-mrtredemo27.vault.azure.net/secrets/nexusvmcert' failed with RequestExcept
ion: 400; desc: {"error":"invalid_request","error_description":"Identity not found"}

marrobi · 2023-05-19T11:14:07Z

The client ID was incorrect, it was id, rather that client_id in the terraform.

marrobi

Need the change making as per comment.

marrobi

LGTM - thanks.

tamirkamara · 2023-05-20T16:27:37Z

/test-shared-services

github-actions · 2023-05-20T16:27:49Z

🤖 pr-bot 🤖

🏃 Running shared service tests: https://github.com/microsoft/AzureTRE/actions/runs/5033123861 (with refid b4903e3e)

(in response to this comment from @tamirkamara)

tamirkamara · 2023-05-21T06:25:52Z

/test-force-approve
nexus tests have passed.

github-actions · 2023-05-21T06:26:05Z

🤖 pr-bot 🤖

✅ Marking tests as complete (for commit ea93b35)

(in response to this comment from @tamirkamara)

tamirkamara added 2 commits May 9, 2023 08:24

nexus config for user id in keyvault extension

d7cceca

make env fix

b480003

Merge branch 'main' into tamirkamara/fix-nexus-keyvault-auth

0766d7b

tamirkamara marked this pull request as ready for review May 10, 2023 16:03

tamirkamara closed this May 11, 2023

tamirkamara reopened this May 11, 2023

reduce azure logging in e2e

c804672

marrobi mentioned this pull request May 16, 2023

Sonatype Nexus VM not successfully deployed #3500

Closed

Merge branch 'main' into tamirkamara/fix-nexus-keyvault-auth

6cd7c82

marrobi requested changes May 19, 2023

View reviewed changes

Comment thread templates/shared_services/sonatype-nexus-vm/terraform/vm.tf

update extension config

98e76aa

marrobi self-requested a review May 19, 2023 12:54

marrobi approved these changes May 19, 2023

View reviewed changes

Merge branch 'main' into tamirkamara/fix-nexus-keyvault-auth

ccb7d3c

changelog

ea93b35

tamirkamara enabled auto-merge (squash) May 21, 2023 06:25

tamirkamara merged commit c916a0e into main May 21, 2023

tamirkamara deleted the tamirkamara/fix-nexus-keyvault-auth branch May 21, 2023 06:36

tamirkamara mentioned this pull request May 28, 2023

Noise in E2E logs #3484

Closed

Conversation

tamirkamara commented May 9, 2023

What is being addressed

How is this addressed

Uh oh!

github-actions Bot commented May 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results

Uh oh!

tamirkamara commented May 10, 2023

Uh oh!

tamirkamara commented May 10, 2023

Uh oh!

tamirkamara commented May 11, 2023

Uh oh!

github-actions Bot commented May 11, 2023

Uh oh!

tamirkamara commented May 11, 2023

Uh oh!

github-actions Bot commented May 11, 2023

Uh oh!

github-actions Bot commented May 16, 2023

Uh oh!

github-actions Bot commented May 16, 2023

Uh oh!

marrobi commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tamirkamara commented May 18, 2023

Uh oh!

github-actions Bot commented May 18, 2023

Uh oh!

tamirkamara commented May 18, 2023

Uh oh!

marrobi commented May 19, 2023

Uh oh!

marrobi commented May 19, 2023

Uh oh!

marrobi commented May 19, 2023

Uh oh!

marrobi commented May 19, 2023

Uh oh!

marrobi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

marrobi left a comment

Choose a reason for hiding this comment

Uh oh!

tamirkamara commented May 20, 2023

Uh oh!

github-actions Bot commented May 20, 2023

Uh oh!

tamirkamara commented May 21, 2023

Uh oh!

github-actions Bot commented May 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented May 9, 2023 •

edited

Loading

marrobi commented May 16, 2023 •

edited

Loading